Computing devices that generate and encode video have been constructed with a pipeline architecture where components cooperate to concurrently perform operations on different video frames. The components typically include a video generating component, a framebuffer, an encoder, and possibly some other components that might multiplex sound data, prepare video frames for network transmission, perform graphics transforms, etc. Typically, the unit of data dealt with by a graphics pipeline has been the video frame. That is, a complete frame fills a framebuffer, then the complete frame is passed to a next component, which may transform the frame and only pass the transformed frame to a next component when the entire frame has been fully transformed.
This frame-by-frame approach may be convenient for the design of hardware and of software to drive the hardware. For example, components of a pipeline can all be driven by the same vsync (vertical sync) signal. However, there can be disadvantages in scenarios that require real-time responsiveness and low latency. As observed only by the instant inventors, the latency from (i) the occurrence of an event that causes graphics (video frames) to start being generated at one device to (ii) the time at which the graphics is displayed at another device, can be long enough to be noticeable. Where the event is a user input to an interactive graphics-generating application such as a game, this latency can cause the application to seem unresponsive or laggy to the user. As only the inventors have appreciated, the time of waiting for a framebuffer to fill with a new frame before the rest of a graphics pipeline can process (e.g., start encoding) the new frame, and the time of waiting for a whole frame to be encoded before a network connection can start video streaming, can contribute to the overall latency.
In addition to the foregoing, to encode video for streaming over a network or a wireless channel, it has become possible to perform different types of encoding on different slices of a same video frame. For example, the ITU's (International Telecommunication Union) H.264/AVC and HEVC/H.265 standards allow for a frame to have some slices that are independently encoded (“ISlices”). An ISlice has no dependency on other parts of the frame or on parts of other frames. The H.264/AVC and HEVC/H.265 standards also allow slices (“PSlices”) of a frame to be encoded based on other slices of a preceding frame with inter-frame prediction and compensation.
When a stream of frames encoded in slices is transmitted on a lossy channel, if an individual Nth slice of one frame is corrupted or dropped, it is possible to recover from that partial loss by encoding the Nth slice of the next frame as an ISlice. However, when an entire frame is dropped or corrupted, a full encoding recovery becomes necessary. Previously, such a recovery would be performed by transmitting an entire Iframe (as used herein, an “Iframe” will refer to either a frame that has only ISlices or a frame encoded without slices, and a “Pframe” will refer to a frame with all PSlices or a frame encoded without any intra-frame encoding). However, as observed only by the present inventors, the transmission of an Iframe can cause a spike in frame size relative to Pframes or frames that have mostly PSlices. This spike can create latency problems, jitter, or other artifacts that can be problematic, in particular for interactive applications such as games.
Described below are techniques related to implementing a graphics pipeline capable of starting to process (e.g., encode) a video frame before the video frame is complete.